System and method for query/modification of linear block address table entries for direct I/O

ABSTRACT

The present invention provides a system, method, and computer program product that enables application instances to pass block mode storage requests directly to a physical I/O adapter without run-time involvement from the local operating system or hypervisor. In one aspect of the present invention, a mechanism is provided for handling user space query and modification operations for querying and modifying attributes of allocations of linear block addresses of a physical storage device. For modification, it is determined if there are sufficient available resources for the modification of the allocation. In addition, it is determined if there are any I/O transactions active on the allocation before performing the modification. Modification of the attributes of the allocation may be performed only if there are sufficient available resources and only if there are no active I/O transactions on the allocation being modified.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates generally to communication protocolsbetween a host computer and an input/output (I/O) Adapter. Morespecifically, the present invention is directed to a system and methodfor enabling user space middleware or applications to pass block modestorage requests directly to a physical I/O adapter without run-timeinvolvement from the local Operating System (OS), or, in a virtualsystem, the local hypervisor.

2. Description of Related Art

Operating systems, according to the present state of the art, do notpermit user space middleware or applications, such as a database, todirectly access persistent storage that is identified through theOperating System's Raw Mode Storage I/O interface or the OperatingSystem's Logical Volume Storage I/O interface. As a result, the userspace middleware must invoke an Operating System (OS) call and incurseveral task switches every time an I/O operation is performed. Thefirst task switch is caused when the middleware or application transfersa storage request to the OS. A second task switch occurs when the OSpasses control back to the user space middleware or application, afterthe OS completes processing the middleware or application storagerequest and passes the storage request to the storage adapter.

A third task switch occurs when the storage adapter completes theassociated block mode I/O storage operations and interrupts theprocessing being performed by an application so that the OS may processthe storage adapter's completion. The final task switch occurs when theOS finishes processing the storage adapter's completion and givescontrol back to the middleware or application that transferred thestorage request to the OS. In addition to these task switches thestorage adapter typically has a single request queue to process workfrom the operating system.

The four task switches described above may be considered wastedprocessor cycles, because all work on the thread being switched isstopped until the task switch is complete. On some servers, the numberof storage operations performed by a user space middleware orapplication program may be quite large. Modern, high-end servers mayhave millions of these operations per second, resulting in severalmillion task switches per second.

SUMMARY OF THE INVENTION

In view of the above, it would be beneficial to have a method, systemand computer program product having computer readable instructions forhandling I/O storage requests in which such task switches are minimized.Moreover, it would be advantageous to have an improved method, system,and computer instructions that enables user space middleware orapplications to pass block mode I/O storage requests directly to aphysical I/O adapter without any run-time involvement from the localOperating System (OS), or, in a virtual system, the local hypervisor. Itwould also be advantageous to have the mechanism apply for InfiniBand,TCP/IP Offload Engines, RDMA (Remote Direct Memory Access) enabled NICs(Network Interface Controllers), iSCSI adapters, iSER (iSCSI Extensionsfor RDMA) adapters, parallel SCSI adapters, Fibre Channel Adapters,Serial Attached SCSI Adapters, ATA Adapters, Serial ATA Adapters, andany other type of storage adapter.

Further, it would be advantageous to have an improved method, system,and computer instructions that enables protection mechanisms to ensurethat block mode storage requests that are directly sent to a physicalI/O adapter from an application instance are only completed to portionsof the storage device that have been previously allocated for out ofuser space I/O with the application instance. Moreover, it would bebeneficial to have a method, system and computer instructions forenabling the creation, modification, querying and deletion of datastructure entries used to facilitate direct block mode I/O operationsbetween an application instance and a physical I/O adapter. In addition,it would be beneficial to have a method, system and computerinstructions for processing user space operations so as to performstorage device resource management and direct I/O operation datastructure management.

The present invention provides a method, computer program product, anddata processing system that enables user space middleware orapplications to pass block mode storage requests directly to a physicalI/O Adapter without run-time involvement from the local Operating System(OS), or, in a virtual system, the local Hypervisor. The mechanismdescribed in this invention applies for InfiniBand Host ChannelAdapters, TCP/IP Offload Engines, RDMA (Remote Direct Memory Access)enabled NICs (Network Interface Controllers), iSCSI adapters, iSER(ISCSI Extensions for RDMA) adapters, parallel SCSI adapters, FibreChannel Adapters, Serial Attached SCSI Adapters, ATA Adapters, SerialATA Adapters, and any other type of storage adapter.

Specifically, the present invention is directed to a mechanism forproviding and using a linear block address (LBA) translation protectiontable (TPT) to control user space, and out of user space, Input/Output(I/O) operations. In one aspect of the present invention, the LinearBlock Address Translation Protection Table (LBATPT) includes an adapterprotection table that has entries for each portion of a storage device,e.g., logical volume, SCSI logical unit number, or the like. Each entrymay include a key instance and protection domain, along with otherprotection table context information, against which I/O requests may bechecked to determine if an application instance that submitted the I/Orequests may access the LBAs identified in the I/O requests. In thisway, only those portions of the storage device that have been allocatedto the application instance may be accessed by the application instance.Moreover, only the application instance for which the portion of thestorage device is allocated may access the portion of the storagedevice.

The LBATPT further includes an LBA table in which are LBA table entriesidentifying the logical block addresses associated with entries in theadapter protection table. The LBA table may be used to map LBAsreferenced in block mode I/O requests to LBAs of the physical storagedevice. The present invention further provides mechanisms for processinguser space operations so as to manage the creation, modification,querying and deletion of entries in the LBA table. Such mechanismsinterface with a memory management interface of the physical I/O adapterso as to allocate, modify, query and de-allocate LBAS associated with anapplication instance.

In addition, the present invention provides mechanisms for processinguser space operations so as to generate work queue entries for passingblock mode I/O operations directly to a physical I/O adapter. Moreover,the present invention provides mechanisms for retrieving completionqueue entries from the physical I/O adapter when a work queue entry hasbeen processed by the physical I/O adapter so as to inform theapplication instance of the completion of the processing.

In one exemplary embodiment of the present invention, a method isprovided in which an input/output (I/O) adapter receives, from anapplication instance, a user space operation requesting modification ofa resource allocation for direct linear block address (LBA) basedinput/output (I/O) between the application instance and a physicalstorage device. The I/O adapter may then modify the resource allocationin a LBA table entry, of an LBA table, associated with the I/O adapterand the application instance and return results of the modification ofthe resource allocation to the application instance. The modifying ofthe resource allocation in the LBA table entry may include using amemory management interface of the I/O adapter to modify the LBA tableentry.

In addition, a determination may be made as to whether there aresufficient resources are available to modify the resource allocation. Ifthere are not sufficient resources available to perform themodification, an error message may be returned.

Moreover, a determination may be made as to whether an I/O transactionis active on the LBA table entry that is being modified by the modifyoperation. The modification may be performed only if there are no I/Otransactions active on the LBA table entry. If there is an I/Otransaction active on the LBA table entry, a timer may be initiated. Adetermination may then be made as to whether a quiescent point isreached before the timer times out. If the quiescent point is reachedbefore the timer times out, the LBA table entry may be modified. Thequiescent point may be, for example, a point at which no I/Otransactions are active on the LBA table entry.

In addition to the above, the LBA table entry may include a pointer toone or more LBAs of the physical storage device. The LBA table entry maybe associated with an adapter protection table entry, in an adapterprotection table, which is used by an application instance to access theone or more LBAs of the physical storage device.

In a further embodiment of the present invention, the I/O adapter mayreceive a second user space operation requesting a query of attributeinformation of the resource allocation. The I/O adapter may thenretrieve attribute information for the resource allocation from an LBAtable entry corresponding to the resource allocation. The retrieval ofthe attribute information for the resource allocation may include usinga memory management interface of the I/O adapter to query the LBA tableentry.

These and other features and advantages of the present invention will bedescribed in, or will become apparent to those of ordinary skill in theart in view of, the following detailed description of the exemplaryembodiments of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features believed characteristic of the invention are setforth in the appended claims. The invention itself, however, as well asa preferred mode of use, further objectives and advantages thereof, willbest be understood by reference to the following detailed description ofan illustrative embodiment when read in conjunction with theaccompanying drawings, wherein:

FIG. 1 is a functional block diagram of a host processor node inaccordance with an exemplary embodiment of the present invention;

FIG. 2 is a diagram illustrating the primary operational elements of ahost processor node used to enable out of user space block mode storageI/O access in accordance with an exemplary embodiment of the presentinvention;

FIG. 3 is a diagram illustrating exemplary control structures used totranslate and protect block mode storage in accordance with an exemplaryembodiment of the present invention;

FIG. 4 is a diagram illustrating exemplary control structures used topass a storage request from a user space middleware or applicationinstance to a storage adapter in accordance with an exemplary embodimentof the present invention;

FIG. 5 is a diagram illustrating exemplary control structures used toassure that a block mode storage I/O request submitted by a user spacemiddleware or application instance is allowed to reference the storagedevice areas referenced in the block mode storage I/O request inaccordance with an exemplary embodiment of the present invention;

FIG. 6 is a flowchart outlining an exemplary operation for handling aninvocation of a user space operation in accordance with an exemplaryembodiment of the present invention;

FIG. 7 is a flowchart outlining an exemplary operation of one exemplaryembodiment of the present invention when an invoked user space operationis a work queue operation requiring generating and processing or workqueue elements;

FIG. 8 is a flowchart outlining an exemplary operation of one exemplaryembodiment of the present invention when performing validation checks todetermine if a work queue entry is valid and may be processed by thephysical I/O adapter;

FIG. 9 is a flowchart outlining an exemplary operation of one exemplaryembodiment of the present invention when the invoked user spaceoperation is a completion queue retrieval process operation;

FIG. 10 is a flowchart outlining an exemplary operation of one exemplaryembodiment of the present invention when creating a LBA entry in an LBAtable in accordance with an exemplary embodiment of the presentinvention;

FIG. 11 is a flowchart outlining an exemplary operation of one exemplaryembodiment of the present invention when processing a user spaceoperation that is a resource modify operation;

FIG. 12 is a flowchart outlining an exemplary operation of one exemplaryembodiment of the present invention when processing a query user spaceoperation; and

FIG. 13 is a flowchart outlining an exemplary operation of one exemplaryembodiment of the present invention when processing a destroy or deleteuser space operation.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention applies to any general or special purpose hostthat uses an I/O adapter, such as a PCI family I/O adapter, virtual I/Oadapter, endpoint device, virtual endpoint device or the like, todirectly attach storage or to attach storage through a network. Thenetwork may consist of endnodes, switches, routers and linksinterconnecting these components. The network links may be FibreChannel, Ethernet, InfiniBand, Advanced Switching Interconnect, anotherstandard storage network interconnect, or a proprietary link that usesproprietary or standard protocols. While the depictions and descriptionhereafter will make reference to particular arrangements of networks andhost nodes, it should be appreciated that the following exemplaryembodiments are only exemplary and modifications to the arrangementsspecifically depicted and described may be made without departing fromthe spirit and scope of the present invention.

It is important to note that the present invention can take the form ofan entirely hardware embodiment, an entirely software embodiment or anembodiment containing both hardware and software elements. In anexemplary embodiment, the present invention is implemented in software,which includes but is not limited to firmware, resident software,microcode, and the like.

Furthermore, the invention can take the form of a computer programproduct accessible from a computer-usable or computer-readable mediumproviding program code for use by or in connection with a computer orany instruction execution system. For the purposes of this description,a computer-usable or computer readable medium can be any apparatus thatcan contain, store, communicate, propagate, or transport the program foruse by or in connection with the instruction execution system,apparatus, or device.

The medium can be an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system (or apparatus or device) or apropagation medium. Examples of a computer-readable medium include asemiconductor or solid state memory, magnetic tape, a removable computerdiskette, a random access memory (RAM), a read-only memory (ROM), arigid magnetic disk and an optical disk. Current examples of opticaldisks include compact disk—read only memory (CD-ROM), compactdisk—read/write (CD-R/W) and DVD.

A data processing system suitable for storing and/or executing programcode will include at least one processor coupled directly or indirectlyto memory elements through a system bus. The memory elements can includelocal memory employed during actual execution of the program code, bulkstorage, and cache memories which provide temporary storage of at leastsome program code in order to reduce the number of times code must beretrieved from bulk storage during execution.

Input/output or I/O devices (including but not limited to keyboards,displays, pointing devices, etc.) can be coupled to the system eitherdirectly or through intervening I/O controllers. Network adapters arecoupled to the system to enable the data processing system to becomecoupled to other data processing systems or remote printers or storagedevices through intervening private or public networks. Modems, cablemodem and Ethernet cards are just a few of the currently available typesof network adapters that may be used with the present invention.

With reference now to the figures, and in particular with reference toFIG. 1, a functional block diagram of a host node is depicted inaccordance with one exemplary embodiment of the present invention. Inthis example, host node 102 includes two processor I/O hierarchies 100and 103 which are interconnected through link 101. For ease of depictingthe elements of the host node 102, only processor I/O hierarchy 100 iscompletely depicted with processor I/O hierarchy 103 having a similar,although not depicted, arrangement of elements as discussed hereafter.

As shown, processor I/O hierarchy 100 includes a processor chip 107which includes one or more processors and their associated caches.Processor chip 107 is connected to memory 112 through a link 108. One ofthe links on the processor chip, such as link 120, connects to a PCIfamily I/O bridge 128. The PCI family I/O bridge 128 has one or more PCIfamily (PCI, PCI-X, PCI-Express, or any future generation of PCI) linksthat are used to connect other PCI family I/O bridges or a PCI familyI/O adapter, such as PCI family adapter 1 145 and PCI family adapter 2144, through a PCI link, such as links 132, 136, and 140. A PCI familyadapter, such as PCI Family Adapter 1 145, may be used to connect to anetwork attached storage 152 through a network link, such as link 156 tonetwork 164, that connects to either a switch or router 160, which inturn connects to the network attached storage 152 via link 158. A PCIfamily adapter, such as PCI family adapter 2 144, may also be used toconnect a direct attached storage device 162 through a link 148.

It is important to note that a PCI Family Adapter, such as PCI FamilyAdapter 1 145 or PCI Family Adapter 2 144, may be integrated with othercomponents on the host node 102. For example, PCI family adapter 1 145or PCI family adapter 2 144, may be integrated with PCI family I/Obridge 128. Another example is that the PCI family adapter, such as PCIfamily adapter 1 145 or PCI family adapter 2 144, may be integrated withprocessor chip 107.

While the exemplary embodiments of the present invention will bedescribed with regard to a PCI family adapter, it should be appreciatedthat the present invention is not limited to this type of adapter.Rather, the physical I/O adapter may be any type of I/O adapterincluding a PCI family adapter, a virtual I/O adapter, an endpointdevice, a virtual endpoint device, a virtual I/O adapter endpointdevice, or the like. One example of a virtual I/O adapter that may beused with the present invention is described in, for example, commonlyassigned and co-pending U.S. patent application Ser. No. 11/065,829entitled “Data Processing System, Method and Computer Program Productfor Creation and Initialization of a Virtual Adapter on a PhysicalAdapter that Supports Virtual Adapter Level Virtualization, filed onFeb. 25, 2005, which is hereby incorporated by reference. Other types ofI/O adapters may be used without departing from the spirit and scope ofthe present invention.

With reference now to FIG. 2, a functional block diagram of systemcomponents associated with one exemplary embodiment of the presentinvention is depicted. In the depicted example, physical I/O adapter 200is an example of a PCI adapter, such as PCI family adapter 1 145 or PCIfamily adapter 2 144 in FIG. 1.

In this example, physical I/O adapter 200, shown in FIG. 2, includes oneset of processing queues (PQs), such as processing queue set 236, andtheir associated processing queue context, such as PQ context 204. Theprocessing queues (PQs) may include, for example, a work queue, such asa send queue and or receive queue, and a completion queue. The workqueue is used to submit linear block address (LBA) based I/O storagerequests, also referred to herein as “block mode I/O storage requests,”directly to the physical I/O adapter. A Linear Block Address (LBA) isthe index of a block, i.e. a fixed size portion of a storage device,from the logical beginning of the storage device. The completion queueis used to communicate completion of a work queue entry back to theapplication instance that submitted the block mode I/O storage request.

The physical I/O adapter 200 also has a translation and protection tablecontext, such as TPT context 208, which is used to contain the contextfor a host resident translation and protection table, such as TPT 232 orTPT 252. The TPT context 208 may also be used to contain the TPT 232 or252 itself or a cache of entries from the host resident TPT 232 or TPT252.

TPT 232 and 252 reside in an Operating System (OS), such as OS 1 220 orOS 2 240. The OS, e.g., OS 1 220 or OS 2 240, may reside above ahypervisor 216, which is software, firmware or a mix of the two thatmanages the partitioning and virtualization of physical hardwareresources and controls the OS execution. The OS may host one or moremiddleware or application instances. In FIG. 2, OS 1 220 is hosting twomiddleware or application instances App 1 224 and App 2 228. Similarly,OS 2 240 is hosting application App 1 224 and App 2 228. The OS runs ona processor, such as processor 212.

A middleware or application instance, such as App 1 224 uses a set ofprocessing queues, such as processing queue set 236, to pass block modeI/O storage requests to the physical I/O adapters. The block mode I/Ostorage requests may be constructed as a SCSI command, as an ATA storagecommand, or the like. When the physical I/O adapter 200 processes ablock mode I/O storage request, the physical I/O adapter 200 uses a keypassed in the block mode I/O storage request to look up an entry in theTPT context 208. If the TPT context 208 is associated with the sameprotection domain as the PQ Context 204, used for the processing queue,then the block mode I/O storage request is processed. Otherwise, theblock mode I/O storage request is completed in error.

Turning next to FIG. 3, an example of a linear block address (LBA)translation and protection table (TPT) is depicted. Two tables are shownin FIG. 3: the adapter protection table 302, and the Linear BlockAddress (LBA) table 322, which together constitute a block storagetranslation and protection table. The adapter protection table 302contains an entry for each logical volume, if logical volume based blockmode I/O storage is used, or SCSI Logical Unit Number (LUN) if SCSI LUNblock mode I/O storage is used. Each of these entries describes theaccess controls and a pointer to the Linear Block Address (LBA) table322, which contains the range of LBAs associated with a correspondingadapter protection table entry. In the depicted example, the adapterprotection table 302 contains entries for each logical volume (LV) andthus, the adapter protection table 302 is an LV adapter protection table302. Entries in the adapter protection table 302 are created in responseto a user space operation for creating a protection table entries in asimilar manner as LBA table entries are created, as discussed in greaterdetail hereafter with reference to FIGS. 6 and 10.

The adapter protection table 302 may be segmented into a set of adapterprotection table segments, such as adapter protection table segment 1304. The segments may be interconnected using several data structures,including a B-tree, a tree made up of pointers in non-leaf nodes andpointers in leaf nodes, simple linked list, or the like. In the depictedexample, adapter protection table segment 1 304 uses a simple linkedlist where the first entry in the table is a pointer to the next tablethat contains adapter protection table entries.

Protection table entry N 320 depicts an example entry in the adapterprotection table segment, such as adapter protection table segment 1304. Each entry in the adapter protection table segment 1 304 contains aset of fields that are used to define that entry. Protection table entryN 320 contains the following fields: Access Controls, Protection Domain,Key Instance, Logical Volume Number, SCSI Identifier Number, SCSILogical Unit Number, LBA Table Size, Sector Size, Length, LBA Tablepointer.

The adapter's S_Key mapping logic 386 performs all the checks of thefields in the protection table entry, e.g., protection table entry N320. Any checks that are not passed by the S_Key mapping logic 386,result in an operation being completed in error. In the event of anerror, the OS may either tear down the middleware or applicationinstance that passed in the operation or take a less drastic measure,such as returning the operation with an error completion.

The Access Controls field describes whether the entry is valid or notand what type of operations may be performed on the entry. The possibleoperations that may be performed on the entry are: Read, Write, andRead/Write. If the block mode I/O storage request passed in by themiddleware or application instance accesses a valid protection tableentry, then the operation passes the valid/invalid check. If the blockmode storage I/O request passed in by the middleware or applicationinstance attempts to perform a Read access operation and the protectiontable entry has the valid bit set, then the operation passes this check.If the block mode storage I/O request passed in by the middleware orapplication instance attempts to perform a Write access operation andthe protection table entry has the Read/Write bit set, then theoperation passes this check.

The Protection Domain field is used to associate an adapter protectiontable entry with a PQ context. That is, if the PQ context, used by themiddleware or application instance to pass in a block mode storage I/Orequests, contains the same value in its protection domain field as theprotection domain field of an adapter protection table entry, then thetwo are associated and the operation passes this check. If there is amismatch between these protection domains in the PQ context and theadapter protection table entry, then the operation fails this check.

The Key Instance is used to compare the storage key passed in a blockmode I/O storage request by the middleware or application instance tothe storage key stored in the adapter protection table entry. If the twomatch, the operation passes this check. If the Key Instance does notmatch the storage key passed in the block mode storage I/O request, thenthe operation does not pass this check.

The storage key, or “S_Key”, has two fields—a first field is an indexinto the adapter protection table and the second field is a key instancewhich is to be compared with a key instance in the adapter protectiontable entry that is pointed to by the first field. When the middlewareor application instance submits a block mode I/O storage request, theadapter uses the first field to obtain an entry from the adapterprotection table. The adapter then compares the key instance inside theadapter protection table entry with the second field that was passed inby the middleware or application instance.

The Logical Volume Number is optional and, if included, it is used tocompare the LV number passed in a block mode storage I/O request by themiddleware or application instance to the LV number stored in theadapter LV protection table entry. If the two match, the operationpasses this check. If the Logical Volume Number does not match the LVnumber passed in by the block mode storage I/O request, then theoperation fails this check.

The SCSI Identifier Number (ID) and SCSI Logical Unit Number (LUN) areused to associate the entry with a specific SCSI device and a specificLUN within that device, respectively.

The LBA Table Size is used to define the maximum number of entries eachLBA table segment that is associated with the adapter protection tableentry, such as LBA table segment 1 324, may contain. The Sector Size isused to define the size of each sector on the disk associated with theadapter protection table entry. The Length field is used to define thetotal length of the set of disk LBAs associated with the adapterprotection table entry.

The LBA Table Pointer of the protection table entry 320 points to one ormore corresponding LBA table entries in the LBA table 322. Thus, withthe LBA Table Pointer field, the linear block addresses associated witha protection table entry in the adapter protection table 302 may beidentified in order to provide access to linear block addresses ofstorage locations on the physical storage device associated with theprocessor queue from which a block mode I/O request is received in theadapter 316.

The LBA table 322 may also be segmented into a set of LBA tablesegments, such as LBA table segment 1 324. The segments may beinterconnected using several data structures, including a B-tree, a treemade up of pointers in non-leaf nodes and pointers in leaf nodes, asimple linked list, or the like. In the depicted example, LBA tablesegment 1 324 uses a simple linked list where the first entry in thetable is a pointer to the next table that contains LBA table entries.

Each entry in the LBA table segment, such as LBA table segment 1 324,describes the range of disk Linear Block Addresses (LBAS) associatedwith that entry. For this description, the entry may use a starting LBAand a length, a starting LBA and an ending LBA, or the like.

The physical I/O adapter, such as adapter 316, may choose to store theentire adapter protection table, a portion of the adapter protectiontable, or none of the adapter protection table. Adapter 316 is shownwith an adapter protection table cache that holds one segment, such ascached protection table segment 1 390.

Similarly, the adapter 316 may choose to store the entire LBA table, aportion of the LBA table, or none of the LBA table. In the depictedexample, adapter 316 is shown with an LBA table cache that holds onesegment, such as cached LV LBA table segment 1 398.

With reference next to FIG. 4, an exemplary diagram of exemplary controlstructures used to pass a block mode I/O storage request for a userspace middleware or application instance to a physical I/O adapter inaccordance with an exemplary embodiment of the present invention isshown. For illustration purposes, a system image, which may be anoperating system, such as Windows XP™, AIX™, Linux™, etc., or a specialpurpose software image, such as a block mode I/O storage server or filemode I/O storage server, for example, is shown with an application thatuses a storage or network adapter to invoke storage operations from astorage device. For purposes of the following description, the terms“system image” and “operating system” may be used interchangeably torefer to a system image, i.e. the current contents of system memory,which may include the operating system and any running applicationinstances.

The system image, such as system image 1 412, has a device driver, suchas adapter driver 440, associated with the storage adapter 420. Theadapter driver 440 may contain a processing queue (PQ) table backingstore 444 which contains copies of the entries in the adapter's PQtable, e.g., system image 1's processing queue table segment 1 400.

When application instance X 432 performs a block mode I/O access, theapplication instance informs the associated adapter 420 through the useof a processing queue (PQ) doorbell 436. For example, PQ 1 Doorbell 436informs the adapter 420 that there is a storage work request in the sendqueue 428 of the processing queue set used to communicate betweenapplication instance X 432 and adapter 420.

The data from PQ 1 Doorbell 436 provides the number of work requeststhat need to be added by an adder 422 to the current number of pendingwork requests in the adapter 420. That is, a block mode I/O request sentgenerated by middleware or an application instance may be comprised of aplurality of actual work requests that are stored in the send queue aswork queue entries. The PQ 1 Doorbell 436 identifies the number of workrequests that are part of the block mode I/O request.

The number of work requests is provided as a PQ Count field that isstored in the associated processing queue table entry PQ N associatedwith the system image, such as cached PQ entry N from system image 1'sPQ Segment 1 424. Once the storage work request is completed, a messageis added to a completion queue 450 which is used to inform theapplication that the work request has been completed.

As shown in FIG. 4, the cached PQ entry N from system image 1's PQsegment 1 424 includes PQ context information including a PQ headaddress, PQ start address, PQ end address, PQ count, and additional PQcontext information. The PQ start address field stores the system memoryaddress of the first work queue entry in the application's processorqueue 428. The PQ End address field stores the last system memoryaddress that is associated with the last work queue entry of theprocessor queue 428. The PQ head address field stores the system memoryaddress of the next processor queue entry that the adapter is going toprocess. The adapter changes the PQ head address as it processesprocessor queue entries in the circular processor queue. The PQ countfield stores the number of processor queue entries that have not beenprocessed by the adapter yet, but have been posted by the applicationinstance 432.

With reference next to FIG. 5, a description of exemplary controlstructures that are used to assure that a block mode I/O storagerequest, submitted by a user space middleware or application instance,is authorized to reference the storage device areas referenced in theblock mode I/O storage request, in accordance with one exemplaryembodiment of the present invention, is provided. FIG. 5 focuses on theprotection of the storage blocks associated with the applicationinstance, by ensuring that only the application instance associated withthose storage blocks is the only application instance that may accessthose storage blocks on the storage devices.

As shown in FIG. 5 a system image 1 500 hosts an application instance X532. This application instance X 532 performs a block mode I/O storagerequest using the mechanism described above with reference to FIG. 4.This mechanism uses a processing queue to submit block mode I/O storagerequests to the desired physical I/O adapter, such as adapter 516. Ablock mode I/O storage work request is put into the send queue 528 whichis a work queue that is part of the processing queue set associated withapplication instance X 532 and adapter 516. The processing queue context517 on the adapter 516, such as that in the cached PQ entry N fromsystem image (SI) 1's PQ Segment 1 524, contains a protection domainfield 518.

When the application X 532 submits a block mode I/O storage request,such as block mode I/O storage request 536, part of the request willcontain an S_Key. The S_Key is used by the adapter 516 to access asegment of the adapter protection table 502. This accessing of a segmentof the adapter protection table 502 may involve accessing a cachedadapter protection table segment 540 if the required segment is presentin the adapter's protection table segment cache 550. If the requiredadapter protection table segment is not present within the adapter'sprotection table segment cache 550, a loading of the required adapterprotection table segment, e.g., adapter protection table segment 504,from the system image 500 into the adapter's protection table segmentcache 550 may be required, for example. Alternatively, the S_Keychecking logic 519 may access the adapter protection table segment,e.g., adapter protection table segment 1 504, in the system image 500directly.

The S_Key is used to lookup an entry in the adapter protection tablesegment 1 504, or cached adapter protection table segment 540,associated with the S_Key. For example, the S_Key may be comparedagainst the key instance fields of the adapter protection table entriesin the adapter protection table segment 504/540 to identify an entrythat matches the S_Key.

S_Key checking logic 519 in the adapter 516 will then perform aprotection domain check to verify that the protection domain coming fromthe PQ context 524 in adapter 516 matches the protection domain in theprotection table entry N 520 that is pointed to by the S_Key in theblock mode I/O storage request 536. Any checks that are not passed bythe S_Key checking logic 519, result in an operation being completed inerror. In such a case, the operating system, e.g., system image 1 500,may either tear down the middleware or application instance, e.g.,application instance X 532, that passed in the operation, or take a lessdrastic measure, such as returning the operation with an errorcompletion.

Assuming that all of the checks previously discussed above are passed,the block mode I/O storage request is processed by the adapter 516 toeither read, write or read/write data to/from the linear block addressesof the physical storage device 560, e.g., the hard disk, referenced bythe entry in the cached LBA table segment 550, or alternatively the LBAtable segment (not shown) associated with the system image 500,corresponding to the adapter protection table entry.

The present invention uses the adapter protection table and LBA table tomanage the “user space” and “out of user space” block mode I/Ooperations. User space is the portion of system memory used to run userapplications. Block mode I/O operations that are performed in “userspace” include operations regarding the creation, modification, queryingand deletion of LBA table entries, the submission and processing of workqueue requests by applications, other I/O operations performed by asystem image, and the like. With regard to the present invention, blockmode I/O operations that are performed “out of user space” includeoperations performed in the I/O adapter to facilitate validation andexecution of I/O requests to physical storage devices.

The data structures and mechanisms described above are used to controlaccess to portions of storage by applications during linear blockaddress (LBA) based direct I/O operations, or simply block mode I/Ooperations, between the application instance and the physical I/Oadapter. The following description provides details regarding the way inwhich resources are allocated, work queue entries are created, andcompletion queue entries are processed in accordance with the mechanismspreviously described above.

FIG. 6 is a flowchart outlining an exemplary operation for handling aninvocation of a user space operation in accordance with an exemplaryembodiment of the present invention. In an exemplary embodiment of thepresent invention, the operations outlined in FIG. 6 are performed by asystem image or operating system in response to an invocation of a userspace operation. While the exemplary embodiments have these operationsbeing performed in a system image or operating system, the presentinvention is not limited to such. Rather, for example, the operationsmay be performed in a user space application, a hypervisor, or the like.

It will be understood that each block, and combination of blocks, of theflowchart illustration in FIG. 6, and the flowchart illustrations insubsequent figures described hereafter, can be implemented by computerprogram instructions. These computer program instructions may beprovided to a processor or other programmable data processing apparatusto produce a machine, such that the instructions which execute on theprocessor or other programmable data processing apparatus create meansfor implementing the functions specified in the flowchart block orblocks. These computer program instructions may also be stored in acomputer-readable memory or storage medium that can direct a processoror other programmable data processing apparatus to function in aparticular manner, such that the instructions stored in thecomputer-readable memory or storage medium produce an article ofmanufacture including instruction means which implement the functionsspecified in the flowchart block or blocks.

Accordingly, blocks of the flowchart illustrations support combinationsof means for performing the specified functions, combinations of stepsfor performing the specified functions and program instruction means forperforming the specified functions. It will also be understood that eachblock of the flowchart illustrations, and combinations of blocks in theflowchart illustrations, can be implemented by special purposehardware-based computer systems which perform the specified functions orsteps, or by combinations of special purpose hardware and computerinstructions.

As shown in FIG. 6, the operation starts with an invocation of a userspace operation (step 610). This invocation may be performed, forexample, by way of a user management interface, an automatedscript/workflow, or the like. The invocation may be made via anapplication instance, the system image, or the like. One example of auser management interface which may perform such an invocation is theraw mode I/O in the Advanced Interactive Executive (AIX) operatingsystem. Other operating systems may have similar interfaces. This usermanagement interface is invoked for management operations like createvolume, destroy volume as well as functional operations such as read orwrite.

A determination is made as to whether the user space operation that isbeing invoked is a resource management operation (step 615). Theoperating system is there to restrict access to the underlying hardwareso that an application cannot access resources associated with anotherapplication. Thus, a resource management operation is one that must beperformed by the operating system because there is no other alternativefor restricting the application's access to the resources it owns.Examples of such operations include create volume, query volume, destroyvolume. A non-resource management operation is one where, through themechanisms of the present invention, a physical adapter can restrict theapplication's access to the resources it owns. Examples of non-resourcemanagement operations are read and write operations.

If the operation is not a resource management operation, then theoperation is a processor queue processing operation. As a result, adetermination is made as to whether the operation is for work queueprocessing, e.g., processing associated with an entry in a send queue(step 620). If so, a work queue entry insertion process is invoked forcreating a work queue entry (step 630). This work queue entry insertionprocess is used to submit work requests to the I/O adapter as discussedpreviously and outlined hereafter in FIG. 7.

If the operation is not for work queue processing, then a completionqueue entry retrieval process is invoked (step 630). The completionqueue entry retrieval process is used to retrieve completion queueentries from the physical I/O adapter for work requests that have beencompleted by the physical I/O adapters, as described in greater detailhereafter.

If the user space operation is a resource management operation (step615), then a determination is made as to whether the operation is aresource query operation (step 640). If the operation is a resourcequery operation, then the system image/operating system retrieves theattributes of the resource from the physical I/O adapter and returns theresults to the element, e.g., system image or application instance, thatinvoked the user space operation (step 645). This operation is used toobtain attribute information from, for example, LBA table entries andadapter protection table entries, as discussed in greater detailhereafter.

If the operation is not a resource query operation, then a determinationis made as to whether the operation is a resource create operation (step650). If the operation is a resource create operation, a determinationis made as to whether the physical I/O adapter has resources availableto allocate to the element invoking the user space operation (step 660).For example, as discussed above, each adapter protection table entry inthe adapter protection table contains a LBA table size, sector size andlength. These parameters may limit the number of resources available bythe adapter for allocation. Thus, the physical I/O adapter may determinethat sufficient resources are not available for allocation to theelement invoking the user space operation.

If there are sufficient resources available to allocate, then theseresources are allocated on the physical I/O adapter and the physical I/Oadapter returns the results of this allocation to the invoking element(step 665). If there are not sufficient resources available to allocate,then an error record may be generated and returned to the elementinvoking the user space operation (step 670).

If the operation is not a resource create operation (step 650), then adetermination is made as to whether the operation is a resource destroyoperation, also referred to herein as a “delete” or “de-allocation”operation (step 675). If the operation is a resource destroy operation,then the resource is destroyed on the physical I/O adapter and theresults of the operation are returned to the element invoking the userspace operation (step 680). If the operation is not a resource destroyoperation, then the operation is a resource modify operation and theattributes of the designated resource are modified on the physical I/Oadapter (step 685). The operation then terminates.

FIG. 7 is a flowchart outlining an exemplary operation of the presentinvention when an invoked user space operation is a work queue operationrequiring generating and processing or work queue elements. Theoperation shown in FIG. 7 corresponds to step 625 in FIG. 6, forexample.

As shown in FIG. 7, the operation starts when the application instanceadds one or more work queue elements to a work queue, e.g., send queue,of a processor queue set associated with the application instance andadapter (step 710). As discussed above, this work queue entry includesan S_Key, a protection domain, an identifier of the I/O operation thatis to be performed, and optionally a logical volume number and SCSI LUN.

A processing queue doorbell message is sent from the applicationinstance to the physical I/O adapter to inform the physical I/O adapterof the newly posted work request (step 715). In one exemplary embodimentof the present invention, the sending of the processor queue doorbellmessage involves performing a programmed I/O write to a doorbell addressassociated with the work queue. As discussed above, the doorbell messageis used to add additional work requests to the processor queue count inthe cached processor queue entry of the physical I/O adapter.

Thereafter, the physical I/O adapter performs validation checks of theinformation stored in the work queue entry or entries against the datastored in the adapter protection table entries (step 720). As discussedabove, these checks may include, for example, checking for matchesbetween protection domains, logical volume numbers, SCSI identificationnumbers, SCSI logical unit numbers, and the like. These checks will bedescribed in greater detail hereafter.

A determination is made as to whether all of the checks are completedsuccessfully (step 725). If all of the checks complete successfully, thephysical I/O adapter uses the LBA table to translate the LBAs referencedwithin the work queue entry and performs LBA containment checks (step730). Because the application instance operates in a different spacethan the storage device, the addresses referenced by the block mode I/Ostorage request generated by the application instance may differ fromthe actual physical addresses of the storage device. The LBA tableentries provide information regarding the actual physical LBAs for thestorage device that are allocated to a particular application instance,as determined from the corresponding adapter protection table entry.Thus, a mapping between the LBAs referenced in the block mode I/Ostorage request, and thus the work queue entries, and the LBAsreferenced in the LBA table may be performed in order to determine theactual physical LBAs to which the block mode I/O operations are to bedirected.

For example, the LBA table pointer in the adapter protection table entrymay be used to access one or more entries in the LBA table thatcorrespond to the adapter protection table entry. From the LBA tableentries corresponding to the adapter protection table entry, the rangeof disk Linear Block Addresses (LBAs) corresponding to the adapterprotection table entry may be identified. These LBAs may then be used tomap the LBAs referenced in the work queue entry to LBAs of the physicalstorage device.

Returning to FIG. 7, a determination is made as to whether the LBAcontainment checks complete successfully (step 735). These LBAcontainment checks are checks to determine if the mapped LBAs in theblock mode I/O operations, and thus the work queue entry, fall withinthe LBAs allocated to the application instance as identified in thecorresponding LBA table entry. For example, if the application instanceis attempting to access portions of the storage device that are notallocated to that application instance, then at least one of the LBAcontainment checks will fail. If any of the validation checks or thecontainment checks are not completed successfully, an error result isgenerated (step 740).

If the validation and containment checks complete successfully, thephysical I/O adapter marks the work queue entry as valid (step 750) andperforms all functions, e.g., read, write, read/write, associated withthe work queue entry (step 755). Thereafter, or after the generation ofan error result in step 740, the physical I/O adapter creates acompletion queue entry associated with the work queue entry and performsa direct memory access (DMA) operation to send the completion queueentry to the application instance (step 760).

A determination is then made as to whether a complete queue event wasrequested (step 765). If so, the physical I/O adapter generates acompletion queue event (step 770) and the operation terminates. That is,after work requests that were posted to the send and receive queues ofthe processor queue are completed, a completion message is put into thecompletion queue and if the application requests it, an event may begenerated.

It is important to note in FIG. 7 that after steps 710 and 715, thesystem image or operating system is not involved in the processing ofthe work queue entry. To the contrary, the adapter performs all of thenecessary operations for performing the validity and containment checks,performing the functions associated with the work queue entry,generating a completion queue entry, and sending the completion queueentry to the host. Thus, the many task switches experienced in knownsystems during an I/O operation, as described in the Background of theInvention above, are avoided by the present invention since it is notnecessary to involve the operating system or system image during theactual checking and processing of the I/O operation after it has beensubmitted by the operating system or system image. The operating systemor system image is only again utilized to retrieve a completion queueentry associated with the work queue entry that was processed, and passthis completion queue entry to the application.

Exemplary validation checks that are performed to determine if a workqueue entry is valid and may be processed by the physical I/O adapterare illustrated in FIG. 8. The validation check operation outlined inFIG. 8 may correspond, for example, to steps 720 and 725 in FIG. 7.

As shown in FIG. 8, the operation starts by retrieving the next workqueue entry for the block mode I/O operation from the work queue, e.g.,send queue (step 810). This work queue entry is then checked against acached or system image resident adapter protection table entry todetermine if the corresponding block mode I/O operation can beperformed. First, the S_Key in the work queue entry is used to lookup anadapter protection table entry corresponding to the S_Key (step 820).For example, the S_Key in the work queue entry is compared against thekey instance in the adapter protection table entries to identify anadapter protection table entry that corresponds to the work queue entry.Not all work queue entries from the same application instance willnecessarily have the same S_Key. Whether or note the same S_Key is usedwith all work queue entries is dependent upon the particularapplication. For example, if the application has only one entry in theadapter protection table, then all reads and writes will go to that sameadapter protection table entry. However, if the application has multipleentries in the protection table, then there will be multiple S_Keys witheach S_Key being unique to each entry.

A determination is made as to whether a valid adapter protection tableentry has been found by the above lookup operation (step 830). If not,an error result is generated and returned (step 840). As mentionedabove, this error result may be, for example, tearing down themiddleware or application instance that passed in the work request thatgenerated the work queue entry, or may take a less drastic measure suchas returning the operation with an error completion.

If a valid adapter protection table entry has been found, a check isthen made as to whether the I/O operation that is to be performed inassociation with the work queue entry is supported by the associatedadapter protection table entry (step 850). For example, the accesscontrols of the adapter protection table entry are compared against theI/O operation identifier in the work queue entry to determine if theadapter protection table entry indicates that the I/O operation may beperformed or not.

If the I/O operation cannot be performed based on the setting of theaccess controls in the adapter protection table entry, then theoperation generates and returns an error result (step 840). If the I/Ooperation can be performed as indicated by the adapter protection tableentry, then a determination is made as to whether the protection domainof the work queue entry corresponds to the protection domain of theadapter protection table entry (step 860). If the protection domains donot match, then the operation generates and returns an error result(step 840). If the protection domains do match, and the work queue entryhas an associated LV number identifier and/or a SCSI LUN identifier,then additional checks of this information are made (steps 870). As withthe previous checks, if these checks result in a mismatch between thework queue entry and the adapter protection table entry, then an errorresult is generated and returned (step 840).

If all checks are passed, the work queue entry is preliminarily markedas a valid work queue entry that may be processed by the physical I/Oadapter (step 880). This preliminary marking of validity only means thatthe work queue entry has passed the first set of validity checks. Asdescribed above, the work queue entry must also pass containment checksbefore being processed by the physical I/O adapter. Following step 880,the operation ends with regard to the validity checks but, as shown inFIG. 7, continues to step 730 or 740 in the overall operation.

It should be appreciated that while FIG. 8 illustrates a series ofchecks that are performed in order to process a block mode I/Ooperation, the present invention is not limited to the particular seriesof checks depicted. To the contrary, the operation outlined in FIG. 8 isonly exemplary and many modifications may be made without departing fromthe spirit and scope of the present invention. For example, the order inwhich the various validity checks are performed may be modified asneeded such that a different series of validity checks is performedhaving a different order of operations.

FIG. 9 is a flowchart outlining an exemplary operation of the presentinvention when the invoked user space operation is a completion queueretrieval process operation. The operation shown in FIG. 9 correspondsto step 630 in FIG. 6, for example.

As shown in FIG. 9, the operation starts by polling the completion queueto determine if there are any completion queue entries ready to beprocessed (step 910). A determination is made as to whether anycompletion queue entries are ready to be processed (step 920). If not, anull result is returned to the user space application (step 930). Ifthere are completion queue entries ready to be processed, the nextcompletion queue entry is returned to the user space application (step940) and the operation terminates.

It should be noted that the above operations described in FIGS. 6-9 areapplicable to LBA based direct I/O operations in both non-virtual andvirtual systems. In a virtual system, the only addition may be theinvocation of the hypervisor, or other virtualization mechanism, by theoperating system or system image to aid in maintaining a continuousrange of virtual LBAs during resource creation, modification, queryingor deletion.

As discussed above, with regard to the operation outlined in FIG. 6, themechanisms of the present invention involve determining whether aninvoked user space operation is directed to the creation, querying,modification or deletion of resource allocations for linear blockaddress based direct I/O between the application and the adapter. Basedon these determinations, various operations for creating, modifying,querying or deleting resource allocations may be invoked by theoperating system or system image. Each of these operations will now bedescribed, with reference to FIGS. 10-13, and with regard to the linearblock address table of the present invention. It should be appreciatedthat the operations shown in FIGS. 10-13 may be performed for virtualand non-virtual systems. Thus, for example, the operations may beperformed to create, modify, query and delete or destroy LBA entriesbased on logical volumes, SCSI identifier, or SCSI logical unit numbers.

Moreover, it should be appreciated that while the operations outlined inFIGS. 10-13 make reference to the creation, modification, querying anddeletion of LBA entries, these same or similar processes may be used togenerate adapter protection table entries as well. That is, in placeswithin the flowcharts shown in FIGS. 10-13 where the LBA table entriesare referenced, these portions of the flowcharts may be modified toreference adapter protection table entries to thereby adapt theseoperations to the creation, modification, querying and deletion ofadapter protection table entries in accordance with an exemplaryembodiment of the present invention.

FIG. 10 is a flowchart outlining an exemplary operation of the presentinvention when creating a LBA entry in an LBA table in accordance withan exemplary embodiment of the present invention. The operation outlinedin FIG. 10 corresponds, for example, to step 665 in FIG. 6.

As shown in FIG. 10, the operation starts by receiving a user spaceoperation that is a request to create one or more LBA entries, i.e.allocate a set of LBAs for direct I/O access by the application instanceand/or system image (step 1010). In response to receiving the creationuser space operation, the operating system or system image uses thephysical I/O adapter's memory management interface to request that thephysical I/O adapter create one or more LBA entries (step 1020). Thememory management interface can be implemented several different ways.For example, the memory management interface may be a queue whereresource management operations can be passed from the system image tothe adapter.

A determination is then made as to whether the I/O adapter hassufficient resources to complete the request (step 1030). Thisdetermination may involve, for example, using the adapter protectiontable entry corresponding to the application instance, as determined bythe S_Key mapping logic, to identify a LBA table segment associated withthe application instance. A determination may then be made, based on theLBA table size, sector size, length, and current available space in theLBA table segment whether there are sufficient resources for allocatingadditional LBAs to the application instance.

For example, each LBA table segment can fit a fixed number of entries.When that number of entries is reached, a new LBA table segment needs tobe added and linked via the LBA table pointer. The operating system, orsystem image, determines if there are free pages available that can bepinned and used to hold a new LBA table segment. If there are free pagesavailable, the operating system or system image, uses those pages, pinsthem, and successfully completes the request; otherwise, the requestfails. LBAS must also be available on the disk for use by theapplication instance. The operating system or system image determines ifthere are LBAs available on the disk that can be used by the applicationand if there are, the operating system or system image uses those LBAs,and references them through the LBA table segment entry; otherwise, therequest fails.

If there are sufficient resources available to allocate the requestedLBAs to the application instance, then a LBA entry is created in the LBAtable segment corresponding to the application instance (step 1040). TheLBA entry identifies the physical storage device LBAs that map to theLBAs requested by the application instance. The physical I/O adapterthen returns the results of the creation user space operation to theapplication instance so as to inform the application instance of theLBAs that may be used by the application instance to perform block modedirect I/O to the physical I/O adapter (step 1050).

If there are not sufficient resources to allocate the requested LBAS,then the physical I/O adapter does not create the LBA entry (step 1060).The resulting error is then returned by the physical I/O adapter to theapplication instance as the results of the creation user space operation(step 1050). The operation then terminates.

FIG. 11 is a flowchart outlining an exemplary operation of the presentinvention when processing a user space operation that is a resourcemodify operation. The operation outlined in FIG. 11 may correspond, forexample, to step 685 of FIG. 6.

As shown in FIG. 11, the operation starts by receiving, from anapplication instance, system image, or the like, a user space operationrequesting modification of one or more LBA entries (step 1110). Thesystem image then uses the physical I/O adapter's memory managementinterface to request that the physical adapter modify one or more LBAentries associated with the application instance or system image (step1120). A determination is made as to whether the physical I/O adapterhas sufficient resources to complete the modification request (step1130). That is, while the protection table entry has a fixed set offields and thus, an insufficient resources scenario will not apply to aprotection table entry after it has been created, the LBA table segmentcan have additional entries added to it and, as described previously,there are scenarios where the LBA table segment may run out ofresources. If there are not sufficient resources available to thephysical I/O adapter to complete the modification request, the physicalI/O adapter returns an error message to the application instanceindicating an inability to complete the modifications (step 1140).

If there are sufficient resources available, a determination is made asto whether there are any active I/O transactions on the LBA entry thatis being modified (step 1150). If there are active I/O transactions onthe LBA entry that is being modified, the physical I/O adapter initiatesa timer and waits for a quiescent point to be reached (step 1160). Thequiescent point is a point at which there are no I/O transactions activeon the LBA entry being modified. This check and waiting for a quiescentpoint are necessary so that modifications are not made to an LBA entrythat would result in corruption of the system due to the active I/Otransactions operating under the previous LBA entry attributes.

A determination is then made as to whether the quiescent point isreached before a timeout of the timer (step 1170). If not, an errormessage is returned to the application instance indicating an inabilityto complete the modifications (step 1140). If a quiescent point isreached before timeout of the timer, the physical I/O adapter modifiesthe attributes of the LBA entry (step 1180) and returns the attributesof the modified resource to the application instance (step 1190). Theoperation then terminates.

FIG. 12 is a flowchart outlining an exemplary operation of the presentinvention when processing a query user space operation. The operationoutlined in FIG. 12 may correspond, for example, to step 645 of FIG. 6.

As shown in FIG. 12, the operation starts by receiving, from anapplication instance, the system image, or the like, a user spaceoperation that requests a query of the attributes of an LBA entry (step1210). In response to receiving this user space operation, the systemimage uses the adapter's memory management interface to request that thephysical I/O adapter query one or more LBA entries (step 1220). Thephysical I/O adapter then returns the attributes of the LBA entry to theapplication instance (step 1230).

FIG. 13 is a flowchart outlining an exemplary operation of the presentinvention when processing a destroy or delete user space operation. Theoperation shown in FIG. 13 corresponds, for example, to step 680 of FIG.6. An LBA table entry may be destroyed or deleted, for example, if theoperating system or system image allows a logical volume to be reducedby the middleware or application instance. This reduction may then causeLBA entries to be destroyed or deleted, for example.

As shown in FIG. 13, the operation starts by receiving a destroy ordelete user space operation (step 1310). In response to receiving thedestroy or delete user space operation, the system image uses thephysical I/O adapter's memory management interface to request that thephysical I/O adapter destroy or delete one or more LBA entries (step1320). A determination is made as to whether an I/O transaction isactive on the LBA entry that is being deleted or destroyed (step 1330).

If an I/O transaction is active on the LBA entry, the physical I/Oadapter initiates a timer and waits for a quiescent point to be reached(step 1340). A determination is then made as to whether the quiescentpoint is reached before the timer times out (step 1350). If not, thephysical I/O adapter creates an error result and returns the errorresult to the application instance (step 1360). If the quiescent pointis reached before the timer times out, or if there is no active I/Otransaction on the LBA entry, the physical I/O adapter destroys ordeletes the existing LBA entry (step 1370) and returns the result to theapplication instance (step 1380). When an LBA entry is destroyed ordeleted by the operating system or system image, an entry is removedfrom the LBA table segment and the LBAs in the disk are freed and madeavailable for use of other applications.

It should be noted that the operations outlined in the flowchartsdescribed above make reference to the operations performed on the LBAtable entries. In addition, changes to the LBA table may further requirechanges to the adapter protection table. For example, for the queryoperation, the operating system or system image will check if themiddleware or application instance that submits the query request isassociated by the adapter protection table entry referenced by the queryoperation. If so, the query is allowed; otherwise, an error is reported.During the creation and modification operations, the middleware orapplication instance requests the Read and Write Access Control fieldvalues and the Length field value that is to be stored in an associatedprotection table entry and the operating system or system image insertsthese values into the appropriate fields of the adapter protection tableentry along with the rest of the fields. During the delete or destroyoperation, the operating system or system image destroys the associatedLBA table segment(s) and then sets the access control fields ofassociated protection table entries to invalid.

Thus, with the present invention, an invoked user space operation ischecked to see if the operation is a resource query, resource create,resource destroy, resource modify, work queue, or completion queueoperation. Based on this determination, corresponding operations forquerying, creating, destroying, and modifying resource allocations, workqueue entries, and completion queue entries are performed. Thus, if anapplication requires resources in order to perform linear block address(LBA) based direct I/O operations, needs to modify an allocation ofresources in order to perform such direct I/O operations, or needs todestroy resource allocations, the present invention provides mechanismsto achieve these purposes. In addition, the application may submit workqueue entries for processing, and process completion queue entries toobtain information regarding work queue entries whose processing by thephysical I/O adapter has been completed. In this way, LBA based directI/O operations, i.e. block mode I/O operations, are managed by themechanisms of the present invention.

Moreover, as illustrated in the exemplary embodiments described above,the present invention provides a plurality of data structures andmechanisms for handling LBA based I/O operations. These data structuresand mechanisms processor queue to linear block address translation usingprotection table access control. This mechanism ensures that only anapplication associated with the portion of the storage device beingaccessed may actually access the portion of the storage device. Aplurality of validation checks, including a key check and a protectiondomain check, are used to maintain this level of security. These checksensure that the application instance is accessing a valid adapterprotection table entry and that the application has permission to accessportions of the storage device associated with a valid protection tableentry.

It should be noted that, while the above mechanisms of the exemplaryembodiments of the present invention make use of the operating system orsystem image to perform a number of operations with regard to thecreation and management of the adapter protection table entries and LBAtable entries, these operations are not generally performed with eachwork request processed by the adapter. That is, the operating system orsystem image is only involved in the setup of the adapter protectiontable entries and the LBA table entries and the registration of theapplication instances/middleware with the associated LBAs. The operatingsystem or system image is not required in order to process each actualwork request submitted by the middleware or application instance sincethe application and adapter can use the transaction protection table andmechanisms described above to process the work requests. As a result,the present invention eliminates the context switches, and theirassociated overhead, required by prior art mechanisms, as explained inthe background of the invention above.

It is important to note that while the present invention has beendescribed in the context of a fully functioning data processing system,those of ordinary skill in the art will appreciate that the processes ofthe present invention are capable of being distributed in the form of acomputer readable medium of instructions and a variety of forms and thatthe present invention applies equally regardless of the particular typeof signal bearing media actually used to carry out the distribution.Examples of computer readable media include recordable-type media, suchas a floppy disk, a hard disk drive, a RAM, CD-ROMs, DVD-ROMs, andtransmission-type media, such as digital and analog communicationslinks, wired or wireless communications links using transmission forms,such as, for example, radio frequency and light wave transmissions. Thecomputer readable media may take the form of coded formats that aredecoded for actual use in a particular data processing system.

The description of the present invention has been presented for purposesof illustration and description, and is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the art. Theembodiment was chosen and described in order to best explain theprinciples of the invention, the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

1. A method, in an input/output (I/O) adapter, comprising: receiving,from an application instance, a user space operation requestingmodification of a resource allocation for direct linear block address(LBA) based input/output (I/O) between the application instance and aphysical storage device; modifying the resource allocation in a LBAtable entry, of an LBA table, associated with the I/O adapter and theapplication instance; and returning results of the modification of theresource allocation to the application instance.
 2. The method of claim1, wherein modifying the resource allocation in the LBA table entryincludes using a memory management interface of the I/O adapter tomodify the LBA table entry.
 3. The method of claim 1, furthercomprising: determining if sufficient resources are available to modifythe resource allocation; and returning an error message if there are notsufficient resources available to modify the resource allocation.
 4. Themethod of claim 1, further comprising: receiving, from the applicationinstance, a second user space operation requesting a query of attributeinformation of the resource allocation; and retrieving attributeinformation for the resource allocation from an LBA table entrycorresponding to the resource allocation.
 5. The method of claim 4,wherein retrieving attribute information for the resource allocationincludes using a memory management interface of the I/O adapter to querythe LBA table entry.
 6. The method of claim 1, further comprising:determining if an I/O transaction is active on the LBA table entry; andmodifying the LBA table entry only if there are no I/O transactionsactive on the LBA table entry.
 7. The method of claim 6, furthercomprising: initiating a timer if there is an I/O transaction active onthe LBA table entry; determining if a quiescent point is reached beforethe timer times out; and modifying the LBA table entry only if thequiescent point is reached before the timer times out.
 8. The method ofclaim 7, wherein the quiescent point is a point at which no I/Otransactions are active on the LBA table entry.
 9. The method of claim1, wherein the LBA table entry includes a pointer to one or more LBAs ofthe physical storage device, and wherein the LBA table entry isassociated with an adapter protection table entry, in an adapterprotection table, which is used by an application instance to access theone or more LBAs of the physical storage device.
 10. A computer programproduct comprising a computer useable medium having a computer readableprogram, wherein the computer readable program, when executed oncomputing device, causes the computing device to: receive, from anapplication instance, a user space operation requesting modification ofa resource allocation for direct linear block address (LBA) basedinput/output (I/O) between the application instance and a physicalstorage device; modify the resource allocation in a LBA table entry, ofan LBA table, associated with the I/O adapter and the applicationinstance; and return results of the modification of the resourceallocation to the application instance.
 11. The computer program productof claim 10, wherein the computer readable program causes the computingdevice to modify the resource allocation in the LBA table entry by usinga memory management interface of the I/O adapter to modify the LBA tableentry.
 12. The computer program product of claim 10, wherein thecomputer readable program further causes the computing device to:determine if sufficient resources are available to modify the resourceallocation; and return an error message is there are not sufficientresources available to modify the resource allocation.
 13. The computerprogram product of claim 10, wherein the computer readable programfurther causes the computing device to: receive, from the applicationinstance, a second user space operation requesting a query of attributeinformation of the resource allocation; and retrieve attributeinformation for the resource allocation from an LBA table entrycorresponding to the resource allocation.
 14. The computer programproduct of claim 13, wherein the computer readable program causes thecomputing device to retrieve attribute information for the resourceallocation by using a memory management interface of the I/O adapter toquery the LBA table entry.
 15. The computer program product of claim 10,wherein the computer readable program further causes the computingdevice to: determine if an I/O transaction is active on the LBA tableentry; and modify the LBA table entry only if there are no I/Otransactions active on the LBA table entry.
 16. The computer programproduct of claim 15, wherein the computer readable program furthercauses the computing device to: initiate a timer if there is an I/Otransaction active on the LBA table entry; determine if a quiescentpoint is reached before the timer times out; and modify the LBA tableentry only if the quiescent point is reached before the timer times out.17. The computer program product of claim 16, wherein the quiescentpoint is a point at which no I/O transactions are active on the LBAtable entry.
 18. The computer program product of claim 10, wherein theLBA table entry includes a pointer to one or more LBAs of the physicalstorage device, and wherein the LBA table entry is associated with anadapter protection table entry, in an adapter protection table, which isused by an application instance to access the one or more LBAs of thephysical storage device.
 19. A data processing system, comprising: ahost system; and a physical storage device coupled to the host system,wherein the host system: receives, from an application instance, a userspace operation requesting modification of a resource allocation fordirect linear block address (LBA) based input/output (I/O) between theapplication instance and a physical storage device; modifies theresource allocation in a LBA table entry, of an LBA table, associatedwith the I/O adapter and the application instance; and returns resultsof the modification of the resource allocation to the applicationinstance.
 20. The data processing system of claim 19, wherein the hostsystem receives, from the application instance, a second user spaceoperation requesting a query of attribute information of the resourceallocation and retrieves attribute information for the resourceallocation from an LBA table entry corresponding to the resourceallocation.